1. Technical Field
The present invention relates to an image processing technique, and more particularly, to a technique for performing resolution conversion such as providing increased resolution in moving pictures.
2. Background Art
Due to spreading of digital video apparatus networks, it has become common to handle digital images in different standard formats in various input/output apparatuses. In particular, there are various kinds of image sizes ranging from a low resolution to an ultra-high resolution. In still pictures, it is common for an imaging element of a consumer digital still camera to have 5 million pixels or more. Currently, there has been introduced a product with more than 10 million pixels, and it has come close to achieving sufficiently high resolution to be used in printing digital pictures.
However, it is anticipated that the use of digital images in the future will be developed into such cases where a part of an actual image is cut out and enlarged or deformed, and texture mapping is carried out for creating a real CG, etc. Further, when used in electronic commercial transactions or medical services, there is also an interactive display usage for allowing an interested region of an image to be enlarge-displayed when presenting the target product or a diseased portion on a display. Further, since high resolution in still pictures has been achieved to a certain extent, it is anticipated that there will be great needs for expanding the resolution in moving pictures in the future (particularly high-resolution moving pictures which are so good that each of the frame images can be used as a still picture as in a movie film).
As described, it is desired to achieve still higher resolution of image digitally. In order to meet this demand, not only the advancement in imaging elements but also sophistication of the digital image processing is essential.
As the application targets of the high-resolution images, there are two fields, e.g. a field of an image input system such as a camera, and a field of image display system such as a television. This application of the present invention regards mainly to an application of the image input system.
As a conventional technique of this field, there is a technique that employs an optical zooming mechanism to obtain a high-resolution still picture with a wide view angle in the end, even though it uses a low-resolution imaging device.
For example, Patent Document 1 discloses a camera that generates a single wide-view-angle/high-resolution still picture by successively superimposing moving pictures, which are filmed by carrying out continuous optical zooming, on an image buffer while considering the zooming ratio. Further, Patent Document 2 discloses a technique for generating a high-resolution image on a master memory, in which: a zoom lens is moved step by step from a maximum zoom position on a telephoto side to a minimum zoom position on a wide angle side; a plurality of images at each position are written to a magnetic tape; and each image is reduced or enlarged to be converted into the images of the same magnification to be superimposed. Furthermore, Patent Document 3 discloses a technique in which a single high-resolution still picture is generated through laminating optical zoom images to each other while aligning the positions, which is composed into a pyramid format of hetero structure.
Moreover, Patent Document 4 discloses a technique for creating a high-resolution moving picture from low-resolution and high-resolution images without using an optical zoom. That is, corresponding points within moving picture of small number of frames captured by a high-resolution and low-speed camera are obtained from the corresponding point information in consecutive moving pictures captured by a low-resolution and high frame rate camera, and high-resolution consecutive frame moving picture is generated form the high-resolution images by using an interpolation technique of the time direction.
Further, Patent Document 5 also discloses a technique for creating a high-resolution moving picture from low-resolution and high-resolution images. In Patent Document 4 described above, a motion is added to a high-resolution still picture to obtain a moving picture. With the technique of Patent Document 5, however, a frame of the low-resolution moving picture and the high-resolution still picture are related by each sample time, and the moving picture is interpolated spatially to achieve high resolution.
Patent Document 1: Japanese Unexamined Patent Publication H11-252428 (FIG. 3)
Patent Document 2: Japanese Unexamined Patent Publication H8-251477 (FIG. 2)
Patent Document 3: U.S. Pat. No. 6,681,056
Patent Document 4: Japanese Unexamined Patent Publication 2003-203237 (FIG. 1)
Patent Document 5: JP Patent Publication No. 3240339 (FIG. 14)
However, the conventional techniques have the following problems.
First, with the technique for generating a high-resolution image using an optical zoom as disclosed in Patent Documents 1-3, there is such a problem that the high-resolution image generated thereby is limited to a till picture. In practice, however, the resolution of a digital moving picture is lower than that of a digital still picture, so that there is a large demand for achieving increased resolution in the digital moving picture.
There are three problems as follows in the conventional techniques regarding the high resolution of moving pictures.
First, when the techniques of Patent Documents 1-3 are applied to a moving picture, it is necessary to execute both a recording process for filming a moving picture by a camera work according to the intention of the shooting person and an image obtaining process for obtaining a partial image by the optical zoom. When both processes are executed in shifted time from each other, relevancy between the high-resolution image and the low-resolution image in each region becomes unclear. Further, in order to find the spatial relevancy for each region of the image, it is necessary to execute the image obtaining process by the optical zoom for each region. For that, it requires repeated actions of carrying out optical zooming by selecting minute regions within the image, which is a complicated work and it is not practical to be done manually.
Second, the technique disclosed in Patent Document 4 is a technique for improving the frame rate of a high-resolution still picture by time interpolation technique, in which a motion is added to a high-resolution still picture with a motion vector. Thus, the image generated thereby is no better than a series of still pictures. Particularly, in a moving picture, there are various scenes that change smoothly due to a change in form, rotary motion of a subject itself, or a camera work such as panning, tilting, and zooming. Therefore, it is difficult to generate all of the high-resolution images by an interpolation technique of the still picture.
Third, the technique disclosed in Patent Document 5 is a technique for creating a spatially high-resolution moving picture from a low-resolution moving picture. Specifically, the moving picture and the still picture are related at dispersed points on the time axis. Thus, for a moving picture frame having no corresponding information, an edge of a similar signal level is searched by using the information of a frame whose relevancy is already known, and pixels to be spatially interpolated are determined considering it as a move of the same subject. Therefore, there is not only a problem of having a heavy load of searching processing, but also having a possibility when there is a change in the distance between the subject or change in form of the subject that it becomes difficult to achieve increased resolution because the moving picture and the still picture cannot be related.